Voice font speaker and prosody interpolation

ABSTRACT

Multi-voice font interpolation is provided. A multi-voice font interpolation engine allows the production of computer generated speech with a wide variety of speaker characteristics and/or prosody by interpolating speaker characteristics and prosody from existing fonts. Using prediction models from multiple voice fonts, the multi-voice font interpolation engine predicts values for the parameters that influence speaker characteristics and/or prosody for the phoneme sequence obtained from the text to spoken. For each parameter, additional parameter values are generated by a weighted interpolation from the predicted values. Modifying an existing voice font with the interpolated parameters changes the style and/or emotion of the speech while retaining the base sound qualities of the original voice. The multi-voice font interpolation engine allows the speaker characteristics and/or prosody to be transplanted from one voice font to another or entirely new speaker characteristics and/or prosody to be generated for an existing voice font.

BACKGROUND

Conventional text-to-speech (TTS) techniques use a single voice font.This voice font is trained with a recording corpus obtained from onevoice talent. The resulting voice font strongly corresponds to theprosody and characteristics used by the voice talent when recording thecorpus. Accordingly, when being recorded, the voice talent must use thesame style and emotion that is desired in the TTS voice.

As the use of TTS becomes more prevalent, the flexibility of the TTSvoice becomes increasingly important in various application scenarios.For example, an interactive application utilizing TTS to communicatewith the user should provide the user with the ability to select frommultiple voice personalities that are able to express rich emotion typesand speaking styles. As TTS applications become more conversational andpersonal, the ability of the TTS application to adapt the speech styleand/or the emotion of the speech of a single voice to match theconversational content is also desirable.

To get recordings covering a variety of emotions and styles for even asingle voice is costly. Obtaining the desirable variety of recordingsfor multiple voices is not only costly, but impracticable. Attempts totransplant an emotion or speaking style from one recording/voice font toother voice fonts using conventional voice adaptation techniques haveresulted in poor quality voice fonts that fail to convey the desiredemotion and/or style and has highlighted the close relationship betweenthe original recording and the emotion and/or style used by the voicetalent. It is with respect to these and other considerations that thepresent invention has been made. Although relatively specific problemshave been discussed, it should be understood that the embodimentsdisclosed herein should not be limited to solving the specific problemsidentified in the background.

BRIEF SUMMARY

This summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription section. This summary is not intended to identify keyfeatures or essential features of the claimed subject matter, nor is itintended to be used as an aid in determining the scope of the claimedsubject matter.

Embodiments of a multi-voice font interpolation engine include a textparser, one or more characteristic predictors, one or morecharacteristic interpolators, and a normalizer. The multi-voice fontinterpolation engine loads, or otherwise receives, multiple voice fontsfrom the voice font repository into memory. A text parser parses thetext to be spoken into a phoneme sequence that, when combined with avoice font, produces computer-generated speech with the sound, style,and emotion specified by the voice font and provides other informationuseful for predicting natural acoustic features. The characteristicpredictors for natural acoustic features, such as a duration predictor,a V/UV predictor, a fundamental frequency (f0) predictor, and a spectrumpredictor use the corresponding parameter prediction models to predictthe characteristic values for each of the source voice fonts. Thecharacteristic interpolators, such as a duration interpolator, a V/UVinterpolator, a fundamental frequency (f0) interpolator, and a spectruminterpolator, employ different weight sets for interpolatingcharacteristics of the multi-voice font from the selectedcharacteristics of each source voice font.

The interpolation method performed by the multi-voice font interpolationengine predicts characteristic values for components of the input text(e.g., each phoneme or frame in the phoneme sequence) using thecharacteristic prediction model supplied by each source voice font.Next, a relative weighting factor for one or more characteristics areassigned to each the source voice fonts contributing to the multi-voicefont. In various embodiments, the sum of each set of weighting factorsis set to one. The multi-voice font interpolation engine interpolatesthe final duration of each input text component by summing the weightedpredicted characteristic values from the source voice fonts andnormalizes the interpolated f0 values for the phoneme sequence. Theinterpolated duration values, the interpolated spectral trajectoryvalues, the interpolated V/UV decisions, and the normalized interpolatedF0 values may be used in a speech synthesis operation that generates asignal usable by an audio output transducer to produce speech using theresulting multi-voice font having the selected speaker and/or prosodycharacteristics and/or saved as a multi-voice font for later use.

BRIEF DESCRIPTION OF THE DRAWINGS

Further features, aspects, and advantages of the present disclosure willbecome better understood by reference to the following figures, whereinelements are not to scale so as to more clearly show the details andwherein like reference numbers indicate like elements throughout theseveral views:

FIG. 1 is a system diagram of one embodiment of a voice fontinterpolation system implemented in a suitable computing environment;

FIGS. 2A-2C collectively form a high-level flowchart of one embodimentof the interpolation method employed by the multi-voice fontinterpolation engine;

FIG. 3 is one embodiment of a tuning tool providing a user interface forthe multi-voice font interpolation engine operating on a client device;

FIG. 4 is a block diagram illustrating one embodiment of the physicalcomponents of a computing device with which embodiments of the presentinvention may be practiced;

FIGS. 5A and 5B are simplified block diagrams of a mobile computingdevice with which embodiments of the present invention may be practiced;and

FIG. 6 is a simplified block diagram of a distributed computing systemin which embodiments of the present invention may be practiced.

DETAILED DESCRIPTION

Various embodiments are described more fully below with reference to theaccompanying drawings, which form a part hereof, and which show specificexemplary embodiments. However, embodiments may be implemented in manydifferent forms and should not be construed as limited to theembodiments set forth herein; rather, these embodiments are provided sothat this disclosure will be thorough and complete, and will fullyconvey the scope of the embodiments to those skilled in the art.Embodiments may be practiced as methods, systems, or devices.Accordingly, embodiments may take the form of a hardware implementation,an entirely software implementation or an implementation combiningsoftware and hardware aspects. The following detailed description is,therefore, not to be taken in a limiting sense.

Embodiments of a multi-voice font interpolation engine are describedherein and illustrated in the accompanying figures. The multi-voice fontinterpolation engine allows the production of computer generated speechwith a wide variety of speaker characteristics and/or prosody byinterpolating speaker characteristics and prosody from existing fonts.Using the prediction models from multiple voice fonts, the multi-voicefont interpolation engine predicts values for the parameters thatinfluence speaker characteristics and/or prosody for the phonemesequence obtained from the text to spoken. For each parameter,additional parameter values are generated by a weighted interpolationfrom the predicted values. Modifying an existing voice font with theinterpolated parameters changes the style and/or emotion of the speechwhile retaining the base sound qualities of the original voice. Themulti-voice font interpolation engine allows the speaker characteristicsand/or prosody to be transplanted from one voice font to another orentirely new speaker characteristics and/or prosody to be generated foran existing voice font.

FIG. 1 is a system diagram of one embodiment of a voice fontinterpolation system implemented in a suitable computing environment.The voice font interpolation system 100 includes a multi-voice fontinterpolation engine 102 running on a client device 104. The multi-voicefont interpolation engine is a computer program for generatinginterpolated voice fonts having the desired speaker characteristics andprosody. The multi-voice font interpolation engine may be implemented invarious forms, such as applications, services, and libraries. Forexample, the multi-voice font interpolation engine may be implemented asa stand-alone application. In other embodiments, the multi-voice fontinterpolation engine may be implemented as a support applicationaccessible through an application programming interface (API) or alibrary or module to provide voice font interpolation to other programsrendering a text-to-speech (TTS) output.

A voice font storage 106 holds a number of existing voice fonts 108. Thevoice font storage may be local storage (e.g., a hard drive or othersecondary storage on the client device) or remote storage (e.g., astorage device accessible over a network). The existing voice fonts areoften, but are not limited to, original voice fonts trained from arecording corpus collected from a voice talent. Each existing voice fonthas a number of associated parameters that define the sound, style, andemotion of the voice used to render the computer-generated speech.Generally, a voice font includes prediction models for the duration,fundamental frequency (f0), the spectral envelope, and thevoiced/unvoiced (V/UV) decision. Duration contributes mostly to therhythm of the voice. The f0 contour contributes to the tone of thevoice. The spectral envelope, the f0 range, and the voiced/unvoiceddecision are more relevant to the speaker characteristics and prosody.In other words, the spectral envelope, the f0 range, and thevoiced/unvoiced parameter primarily control the style and emotion of thespeech produced using the voice font. The existing voice fonts representdiverse emotions and speaking styles.

The multi-voice font interpolation engine includes a text parser 110, aduration interpolator 112, an f0 interpolator 114, a normalizer 116, aV/UV decision interpolator 118, a spectrum interpolator 120, and. Themulti-voice font interpolation engine loads, or otherwise receives,multiple voice fonts from the voice font repository into memory. Thetext parser parses the input text 122 to be spoken into a phonemesequence that, when combined with a voice font, producescomputer-generated speech with the sound, style, and emotion specifiedby the voice font. In various embodiments, the text parser performsadditional functions, such as, but not limited to, identifying parts ofspeech, phase segmentation, and semantic components in the input text.The additional information provided by the text parser is useful forpredicting natural acoustic features, such as, but not limited to,duration, V/UV, f0, and spectrum characteristics.

The duration interpolator, the V/UV interpolator, the f0 interpolator,and the spectrum interpolator use the corresponding parameter predictionmodels to predict the values for each of the loaded voice fonts. Themulti-voice font interpolation engine employs different weight sets forinterpolating the four characteristics. In various embodiments, threeweight sets are used. The first weight set 124 a is associated withduration (i.e., the duration weights). The second weight set 124 b isassociated with f0 (i.e., the f0 weights). Each weight set includes aweighting factor for each of the voice fonts used by the multi-voicefont interpolation engine. The third weight set 124 c is associated withthe spectrum (i.e., the spectrum weights) and, also, the V/UV decisionbecause the V/UV decision is closely tied (i.e., directly proportional)to the spectral trajectory and independently varying the values tends tosignificantly reduce voice quality. For each increment of the inputtext, the interpolated value is the sum of the products of the predictedvalues and the associated weighting factor for the voice font and summedto produce an interpolated parameter value. Adjusting the weight givento each voice font influencing the interpolated values alters thespeaker characteristics and/or prosody of the computer-generated speech.

The normalizer normalizes the interpolated f0 values using interpolatedupper and lower limits for the f0 range of the predicted f0 values andthe interpolated f0 values. In the illustrated embodiment, themulti-voice font interpolation engine includes a voice encoder (i.e.,vocoder) 126 that renders the input text as speech using theinterpolated values. In various embodiments, the computer-generatedspeech is played through the audio output transducer (i.e., speaker) 128of the client device.

Together, the voice fonts and the weight sets that produce thecomputer-generated speech having the interpolated speakercharacteristics and/or prosody defines a multi-voice font. Generally,source voice fonts most closely resembling the desired speakercharacteristic and/or prosody are selected and the weights are tuned toapproach the desired voice. In various embodiments, the multi-voice fontinterpolation engine saves the multi-voice font as a configuration filespecifying the source voice fonts and the associated weighting factorsin the three weight sets. In some embodiments, the configuration filemay be part of a wrapper that includes the source voice fonts. In otherembodiments, the source voice fonts are stored separately and loadedfrom references in the configuration file.

FIGS. 2A-2C collectively form a high-level flowchart of one embodimentof the interpolation method employed by the multi-voice fontinterpolation engine. The interpolation method 200 begins with a sourcevoice font loading operation 202 that loads multiple source voice fontsinto memory. The source voice fonts provide the reference parametersused from which a multi-voice font is interpolated. A text inputoperation 204 receives the text that is to be converted into speech. Atext parsing operation 206 builds a phoneme sequence from the inputtext.

A voice font characteristic prediction operation 208 predictscharacteristic values for components (i.e., linguistic units) of theinput text (e.g., each phoneme or frame in the phoneme sequence) usingthe characteristic prediction model supplied by each source voice font.In various embodiments the voice font characteristic predictionoperation 208 includes a duration prediction operation 208 a, a V/UVdecision prediction operation 208 b, an f0 prediction operation 208 c,and a spectrum prediction operation 208 d. The duration predictionoperation 208 a predicts the duration value for each phoneme in thephoneme sequence using the duration prediction model supplied by eachsource voice font. The V/UV decision prediction operation 208 b predictsthe V/UV probability for each phoneme in the phoneme sequence using theV/UV decision prediction model supplied by each source voice font. It isnot necessary to make the actual V/UV decision for the phonemes usingeach source voice font as the final V/UV decision will be made from theinterpolated V/UV probability value for the phoneme. The f0 predictionoperation 208 c predicts the f0 value for each frame using the f0prediction model supplied by each source voice font. Each framerepresents a fixed length of time. The spectrum prediction operation 208d predicts the spectral trajectory value for each frame using thespectrum prediction model supplied by each source voice font. In theillustrated embodiment, some of the prediction operations are shown asoccurring in parallel branches because the spectral trajectoryprediction is not dependent on the V/UV decision or f0 predictions.

A characteristic weight setting operation 210 assigns a relativeweighting factor for one or more characteristics to each of the sourcevoice fonts contributing to the multi-voice font. In variousembodiments, the characteristic weight setting operation 210 includes aduration weight setting operation 210 a, an f0 weight setting operation210 b, and a spectrum weight setting operation 210 c. The durationweight setting operation 210 a assigns a relative duration weightingfactor w_(j) ^(d) to each the source voice fonts contributing to themulti-voice font. The f0 weight setting operation 210 b assigns the f0weighting factor w_(j) ^(f) to each the source voice fonts contributingto the multi-voice font. The spectrum weight setting operation 210 cassigns the relative spectrum weighting factor w_(j) ^(s) to each thesource voice fonts contributing to the multi-voice font. In variousembodiments, the sum of each set of weighting factors is set to one, asmathematically expressed in the following equations:

$\begin{matrix}{{\sum\limits_{j = 1}^{N}w_{j}^{d}} = 1} & (1) \\{{\sum\limits_{j = 1}^{N}w_{j}^{f}} = 1} & (2) \\{{\sum\limits_{j = 1}^{N}w_{j}^{s}} = 1} & (3)\end{matrix}$

where j is the index of the source voice font and N is the total numberof source voice fonts.

The duration weighting factors w_(j) ^(d) and the f0 weighting factorsw_(j) ^(f) primarily control the prosody of the multi-voice font whilethe spectrum weighting factors w_(j) ^(s) to primarily control thespeaker characteristics. The duration, f0, and spectrum weightingfactors may be independently controlled to achieve the desired styleand/or emotion effect in the multi-voice font.

A characteristic interpolation operation 214 interpolates the finalduration of each input text component by summing the weighted predictedcharacteristic values from the source voice fonts. In variousembodiments, the characteristic interpolation operation 214 includes aduration interpolation operation 214 a, a spectral trajectoryinterpolation operation 214 b, a V/UV decision interpolation operation214 c, and an f0 interpolation operation 214 d.

The duration interpolation operation 214 a interpolates the finalduration of each phoneme by summing the weighted predicted durationvalues from the source voice fonts. In other words, durationinterpolation operation sums the product of the duration valuespredicted by each source voice font duration prediction model and theduration weighting factor assigned to the source voice font for eachphoneme. Mathematically, the interpolated duration of the i-th phonemefrom the input text is expressed as:

$\begin{matrix}{{{dur}(i)} = {\sum\limits_{j = 1}^{N}{w_{j}^{d}{{dur}_{j}(i)}}}} & (4)\end{matrix}$

where j is the index of source voice font, N is the number of sourcevoice fonts, w_(j) ^(d) is the duration weight for j-th voice font, anddur_(j)(i) is the duration of i-th phoneme predicted by j-th durationprediction model.

The spectral trajectory interpolation operation 214 b interpolates thefinal spectral trajectory of each frame of the phoneme sequence. Invarious embodiments, the spectral trajectory for each source voice fontis predicted using the associated prediction models and theninterpolated using the spectrum weighting factors. In other words, thespectral trajectory interpolation operation sums the product of thespectral trajectory values predicted by each source voice font spectraltrajectory prediction model and the spectrum weighting factor assignedto the source voice font. Mathematically, the interpolated spectraltrajectory of f-th frame from the input text is expressed as:

$\begin{matrix}{{{spec}(f)} = {\sum\limits_{j = 1}^{N}{w_{j}^{s}{{spec}_{j}(f)}}}} & (5)\end{matrix}$

where j is the index of source voice font, N is the number of sourcevoice fonts, w_(j) ^(s) is the spectrum weight for j-th voice font, andspec_(j)(f) is the spectral trajectory of the f-th frame predicted byj-th f0 spectral trajectory prediction model. In other embodiments, thevalues of the underlying characteristic prediction models for eachsource voice font are interpolated first using the correspondingweighting factors to generate an interpolated spectrum prediction modelthat is used to calculate the interpolated spectral trajectory.

The V/UV decision interpolation operation 214 c interpolates whethereach phoneme is voiced or unvoiced by comparing the combined weightedpredicted V/UV probability values from the source voice fonts to athreshold. In other words, the V/UV decision interpolation operationsums the product of the V/UV probability values predicted by each sourcevoice font V/UV decision prediction model and the spectrum weightingfactor assigned to the source voice font for each phoneme and comparesthe sum to a reference value (e.g., a threshold). If the sum is greaterthan or equal to the threshold, the phoneme is voiced in the multi-voicefont. Otherwise, the phoneme is unvoiced in the multi-voice font.Although described as a threshold, alternate logic for making the V/UVdecision may be used without departing from the scope and spirit of theinvention. For example, the reference value may be treated as a ceilingwith sums less than the ceiling indicating that the phoneme is voiced inthe multi-voice font. Mathematically, the V/UV decision of i-th phonemefrom the input text is expressed as:

$\begin{matrix}{{{uv}(i)} = \left\{ \begin{matrix}1 & {{{if}\mspace{14mu} {\sum\limits_{j = 1}^{N}{w_{j}^{s}{{uv}_{j}(i)}}}} \geq {threshold}} \\0 & {else}\end{matrix} \right.} & (6)\end{matrix}$

where j is the index of source voice font, N is the number of sourcevoice fonts, w_(j) ^(s) is the spectrum weight for j-th voice font, anduv_(j)(i) is the V/UV probability for the i-th phoneme predicted by j-thV/UV decision prediction model.

The f0 interpolation operation 214 d interpolates the final f0 value foreach frame of the phoneme sequence by summing the weighted predicted f0values from the source voice fonts. In other words, the f0 interpolationoperation sums the product of the f0 values predicted by each sourcevoice font f0 prediction model and the f0 weighting factor assigned tothe source voice font. Mathematically, the interpolated f0 of f-th frameis expressed as:

$\begin{matrix}{{f\; 0(f)} = {\sum\limits_{j = 1}^{N}{w_{j}^{f}f\; 0_{j}(f)}}} & (7)\end{matrix}$

where j is the index of source voice font, N is the number of sourcevoice fonts, w_(j) ^(d) is the f0 weight for j-th voice font, andf0_(j)(f) is the f0 of the f-th frame predicted by j-th f0 predictionmodel.

An f0 normalization operation 216 normalizes the interpolated f0 valuesfor the phoneme sequence. In order to normalize the interpolated f0values, the upper and lower limits of the target f0 range and theinterpolated f0 range are interpolated using weighted estimates of upperand lower limits for the f0 range for each source voice font. In variousembodiments, the f0 normalization operation includes an f0 range limitestimation operation 216 a that estimates the upper and lower limits ofthe target f0 range based on the values of the predicted f0 ranges. Forexample, the estimated upper and lower limits of the final f0 range maybe calculated as the average upper and lower limits of the f0 rangespredicted for each of the source voice fonts.

A target f0 limit interpolation operation 216 b interpolates the upperand lower limits of the target f0 range using the spectrum weight. Aninterpolated f0 limit interpolation operation 216 c interpolates theupper and lower limits of the interpolated f0 range are interpolatedusing the f0 weight. The upper limit f0^(u) and the lower limit f0^(b)of the target f0 range and the upper limit f0^(u′) and the lower limitf0^(b′) of the interpolated f0 range are mathematically expressed as:

$\begin{matrix}{{f\; 0^{u}} = {\sum\limits_{j = 1}^{N}{w_{j}^{s}f\; 0_{j}^{u}}}} & (8) \\{{f\; 0^{b}} = {\sum\limits_{j = 1}^{N}{w_{j}^{s}f\; 0_{j}^{b}}}} & (9) \\{{f\; 0^{u^{\prime}}} = {\sum\limits_{j = 1}^{N}{w_{j}^{f}f\; 0_{j}^{u}}}} & (10) \\{{f\; 0^{b^{\prime}}} = {\sum\limits_{j = 1}^{N}{w_{j}^{f}f\; 0_{j}^{b}}}} & (11)\end{matrix}$

where j is the index of source voice font, N is the number of sourcevoice fonts, w_(j) _(s) is the spectrum weight for j-th voice font,w_(j) ^(d) is the f0 weight for j-th voice font where f0_(j) ^(u) andf0_(j) ^(b) are the estimated upper and lower limits for j-th font thathave been determined in advance. Finally, a normalized value calculationoperation 216 d calculates and normalizes the interpolated f0 valuesusing the two pairs of upper and lower limits for the f0 range. Thefunction for the normalized fundamental frequency F0 is mathematicallyexpressed as:

$\begin{matrix}{{F\; 0(f)} = {{\frac{{f\; 0(f)} - {f\; 0^{b^{\prime}}}}{{f\; 0^{u^{\prime}}} - {f\; 0^{b^{\prime}}}} \times \left( {{f\; 0^{u}} - {f\; 0^{b}}} \right)} + {f\; 0^{b}}}} & (12)\end{matrix}$

The interpolated duration values, the interpolated spectral trajectoryvalues, the interpolated V/UV decisions, and the normalized interpolatedF0 values are used in a speech synthesis operation 218 that generates asignal usable by an audio output transducer to produce speech using theresulting multi-voice font having the selected speaker and/or prosodycharacteristics. A multi-voice font storage operation 220 saves theresulting multi-voice font for reuse.

FIG. 3 is one embodiment of a tuning tool providing a user interface forthe multi-voice font interpolation engine operating on a client device.In the illustrated embodiment, the user interface 300 is displayed onthe display screen 302 of the client device 104, which is represented bya tablet or other hand held computing device. A source voice fontselection control (e.g., button) 304 allows selected source voice fontsto be loaded. One or more text input controls allows the entry of textto be converted to speech using any of the source voice fonts or themulti-voice font derived from the source voice fonts. In variousembodiments, the text input controls may include an immediate text inputcontrol 306 a that allows immediate entry of input text (e.g., a textbox) or a saved text input control 306 b that allows previously savedtext to be loaded for use as the input text.

One or more source voice font weight selection controls (e.g., sliders)308 allow the relative weights assigned to various characteristics(e.g., duration, spectrum, or f0 weights) of each source voice font tobe adjusted for use as described in the interpolation method 200. Thevarious characteristics may be independently adjusted by accessing thecorresponding characteristic selection control (e.g., tabs) 310. One ormore multi-voice font property controls (e.g. sliders) 312 allow theoverall properties (e.g., volume, speech rate, pitch level, or pitchrange) to be adjusted.

Rendering font selection controls (e.g., buttons) 314 allows theselection of the voice font used to render the input text ascomputer-generated speech. Playback controls (e.g., play, pause, andstop buttons) 316 allow the input text to be rendered ascomputer-generated speech using any of the source voice fonts or theinterpolated multi-voice font. The rendered speech may be played via theaudio output transducer (i.e., speaker) 126 of the client device.

A font save control (e.g., button) 318 allows the multi-voice font to besaved for reuse. A speech save control (e.g., button) 320 allows thecomputer-generated speech rendered using the multi-voice font to besaved as an audio file.

The subject matter of this application may be practiced in a variety ofembodiments as systems, devices, and other articles of manufacture or asmethods. Embodiments may be implemented as hardware, software, computerreadable media, or a combination thereof. The embodiments andfunctionalities described herein may operate via a multitude ofcomputing systems including, without limitation, desktop computersystems, wired and wireless computing systems, mobile computing systems(e.g., mobile telephones, netbooks, tablet or slate type computers,notebook computers, and laptop computers), hand-held devices,multiprocessor systems, microprocessor-based or programmable consumerelectronics, minicomputers, and mainframe computers.

User interfaces and information of various types may be displayed viaon-board computing device displays or via remote display unitsassociated with one or more computing devices. For example, userinterfaces and information of various types may be displayed andinteracted with on a wall surface onto which user interfaces andinformation of various types are projected. Interaction with themultitude of computing systems with which embodiments of the inventionmay be practiced include, keystroke entry, touch screen entry, voice orother audio entry, gesture entry where an associated computing device isequipped with detection (e.g., camera) functionality for capturing andinterpreting user gestures for controlling the functionality of thecomputing device, and the like.

FIGS. 4 and 5 and the associated descriptions provide a discussion of avariety of operating environments in which embodiments of the inventionmay be practiced. However, the devices and systems illustrated anddiscussed are for purposes of example and illustration and are notlimiting of a vast number of computing device configurations that may beutilized for practicing embodiments of the invention described above.

FIG. 4 is a block diagram illustrating physical components (i.e.,hardware) of a computing device 400 with which embodiments of theinvention may be practiced. The computing device components describedbelow may be suitable for embodying computing devices including, but notlimited to, a personal computer, a tablet computer, a surface computer,and a smart phone, or any other computing device discussed herein. In abasic configuration, the computing device 400 may include at least oneprocessing unit 402 and a system memory 404. Depending on theconfiguration and type of computing device, the system memory 404 maycomprise, but is not limited to, volatile storage (e.g., random accessmemory), non-volatile storage (e.g., read-only memory), flash memory, orany combination of such memories. The system memory 404 may include anoperating system 405 and one or more program modules 406 suitable forrunning software applications 420 such as the multi-voice fontinterpolation engine 102 or the multi-voice font tuning tool 300. Forexample, the operating system 405 may be suitable for controlling theoperation of the computing device 400. Furthermore, embodiments of theinvention may be practiced in conjunction with a graphics library, otheroperating systems, or any other application program and is not limitedto any particular application or system. This basic configuration isillustrated by those components within a dashed line 408. The computingdevice 400 may have additional features or functionality. For example,the computing device 400 may also include additional data storagedevices (removable and/or non-removable) such as, for example, magneticdisks, optical disks, or tape. Such additional storage is illustrated bya removable storage device 409 and a non-removable storage device 410.

As stated above, a number of program modules and data files may bestored in the system memory 404. While executing on the processing unit402, the software applications 420 may perform processes including, butnot limited to, one or more of the stages of the interpolation method200. Other program modules that may be used in accordance withembodiments of the present invention may include electronic mail andcontacts applications, word processing applications, spreadsheetapplications, database applications, slide presentation applications,drawing applications, etc.

Furthermore, embodiments of the invention may be practiced in anelectrical circuit comprising discrete electronic elements, packaged orintegrated electronic chips containing logic gates, a circuit utilizinga microprocessor, or on a single chip containing electronic elements ormicroprocessors. For example, embodiments of the invention may bepracticed via a system-on-a-chip (SOC) where each or many of theillustrated components may be integrated onto a single integratedcircuit. Such an SOC device may include one or more processing units,graphics units, communications units, system virtualization units andvarious application functionality all of which are integrated (or“burned”) onto the chip substrate as a single integrated circuit. Whenoperating via an SOC, the functionality described herein with respect tothe software applications 420 may be operated via application-specificlogic integrated with other components of the computing device 400 onthe single integrated circuit (chip). Embodiments of the invention mayalso be practiced using other technologies capable of performing logicaloperations such as, for example, AND, OR, and NOT, including but notlimited to mechanical, optical, fluidic, and quantum technologies. Inaddition, embodiments of the invention may be practiced within a generalpurpose computer or in any other circuits or systems.

The computing device 400 may also have one or more input device(s) 412such as a keyboard, a mouse, a pen, a sound input device, a touch inputdevice, etc. The output device(s) 414 such as a display, speakers, aprinter, etc. may also be included. The aforementioned devices areexamples and others may be used. The computing device 400 may includeone or more communication connections 416 allowing communications withother computing devices 418. Examples of suitable communicationconnections 416 include, but are not limited to, RF transmitter,receiver, and/or transceiver circuitry; universal serial bus (USB),parallel, and/or serial ports.

The term computer readable media as used herein may include computerstorage media. Computer storage media may include volatile andnonvolatile, removable and non-removable media implemented in any methodor technology for storage of information, such as computer readableinstructions, data structures, or program modules. The system memory404, the removable storage device 409, and the non-removable storagedevice 410 are all examples of computer storage media (i.e., memorystorage). Computer storage media may include random access memory (RAM),read only memory (ROM), electrically erasable read-only memory (EEPROM),flash memory or other memory technology, compact disc read only memory(CD-ROM), digital versatile disks (DVD) or other optical storage,magnetic cassettes, magnetic tape, magnetic disk storage or othermagnetic storage devices, or any other article of manufacture which canbe used to store information and which can be accessed by the computingdevice 400. Any such computer storage media may be part of the computingdevice 400.

FIGS. 5A and 5B illustrate a mobile computing device 500 with whichembodiments of the invention may be practiced. Examples of suitablemobile computing devices include, but are not limited to, a mobiletelephone, a smart phone, a tablet computer, a surface computer, and alaptop computer. In a basic configuration, the mobile computing device500 is a handheld computer having both input elements and outputelements. The mobile computing device 500 typically includes a display505 and one or more input buttons 510 that allow the user to enterinformation into the mobile computing device 500. The display 505 of themobile computing device 500 may also function as an input device (e.g.,a touch screen display). If included, an optional side input element 515allows further user input. The side input element 515 may be a rotaryswitch, a button, or any other type of manual input element. Inalternative embodiments, mobile computing device 500 may incorporatemore or less input elements. For example, the display 505 may not be atouch screen in some embodiments. In yet another alternative embodiment,the mobile computing device 500 is a portable phone system, such as acellular phone. The mobile computing device 500 may also include anoptional keypad 535. Optional keypad 535 may be a physical keypad or a“soft” keypad generated on the touch screen display. In variousembodiments, the output elements include the display 505 for showing agraphical user interface, a visual indicator 520 (e.g., a light emittingdiode), and/or an audio transducer 525 (e.g., a speaker). In someembodiments, the mobile computing device 500 incorporates a vibrationtransducer for providing the user with tactile feedback. In yet anotherembodiment, the mobile computing device 500 incorporates input and/oroutput ports, such as an audio input (e.g., a microphone jack), an audiooutput (e.g., a headphone jack), and a video output (e.g., a HDMI port)for sending signals to or receiving signals from an external device.

FIG. 5B is a block diagram illustrating the architecture of oneembodiment of a mobile computing device. That is, the mobile computingdevice 500 can incorporate a system (i.e., an architecture) 502 toimplement some embodiments. In one embodiment, the system 502 isimplemented as a smart phone capable of running one or more applications(e.g., browsers, e-mail clients, notes, contact managers, messagingclients, games, and media clients/players). In some embodiments, thesystem 502 is integrated as a computing device, such as an integratedpersonal digital assistant (PDA) and wireless phone.

One or more application programs 565 may be loaded into the memory 562and run on or in association with the operating system 564. Examples ofthe application programs include phone dialer programs, e-mail programs,personal information management (PIM) programs, word processingprograms, spreadsheet programs, Internet browser programs, messagingprograms, and so forth. The system 502 also includes a non-volatilestorage area 568 within the memory 562. The non-volatile storage area568 may be used to store persistent information that should not be lostif the system 502 is powered down. The application programs 565 may useand store information in the non-volatile storage area 568, such ase-mail or other messages used by an e-mail application, and the like. Asynchronization application (not shown) also resides on the system 502and is programmed to interact with a corresponding synchronizationapplication resident on a host computer to keep the information storedin the non-volatile storage area 568 synchronized with correspondinginformation stored at the host computer. As should be appreciated, otherapplications may be loaded into the memory 562 and run on the mobilecomputing device 500, including software applications 420 describedherein.

The system 502 has a power supply 570, which may be implemented as oneor more batteries. The power supply 570 might further include anexternal power source, such as an AC adapter or a powered docking cradlethat supplements or recharges the batteries.

The system 502 may also include a radio 572 that performs the functionof transmitting and receiving radio frequency communications. The radio572 facilitates wireless connectivity between the system 502 and theoutside world via a communications carrier or service provider.Transmissions to and from the radio 572 are conducted under control ofthe operating system 564. In other words, communications received by theradio 572 may be disseminated to the application programs 565 via theoperating system 564, and vice versa.

The visual indicator 520 may be used to provide visual notifications,and/or an audio interface 574 may be used for producing audiblenotifications via the audio transducer 525. In the illustratedembodiment, the visual indicator 520 is a light emitting diode (LED) andthe audio transducer 525 is a speaker. These devices may be directlycoupled to the power supply 570 so that when activated, they remain onfor a duration dictated by the notification mechanism even though theprocessor 560 and other components might shut down for conservingbattery power. The LED may be programmed to remain on indefinitely untilthe user takes action to indicate the powered-on status of the device.The audio interface 574 is used to provide audible signals to andreceive audible signals from the user. For example, in addition to beingcoupled to the audio transducer 525, the audio interface 574 may also becoupled to a microphone to receive audible input, such as to facilitatea telephone conversation. In accordance with embodiments of the presentinvention, the microphone may also serve as an audio sensor tofacilitate control of notifications, as will be described below. Thesystem 502 may further include a video interface 576 that enables anoperation of an on-board camera 530 to record still images, videostreams, and the like.

A mobile computing device 500 implementing the system 502 may haveadditional features or functionality. For example, the mobile computingdevice 500 may also include additional data storage devices (removableand/or non-removable) such as, magnetic disks, optical disks, or tape.Such additional storage is illustrated by the non-volatile storage area568.

Data/information generated or captured by the mobile computing device500 and stored via the system 502 may be stored locally on the mobilecomputing device 500, as described above, or the data may be stored onany number of storage media that may be accessed by the device via theradio 572 or via a wired connection between the mobile computing device500 and a separate computing device associated with the mobile computingdevice 500, for example, a server computer in a distributed computingnetwork, such as the Internet. As should be appreciated suchdata/information may be accessed via the mobile computing device 500 viathe radio 572 or via a distributed computing network. Similarly, suchdata/information may be readily transferred between computing devicesfor storage and use according to well-known data/information transferand storage means, including electronic mail and collaborativedata/information sharing systems.

FIG. 6 illustrates one embodiment of the architecture of a system forproviding multi-voice font interpolation functionality to one or moreclient devices, as described above. Content developed, interacted with,or edited in association with the software applications 420 may bestored in different communication channels or other storage types. Forexample, various documents may be stored using a directory service 622,a web portal 624, a mailbox service 626, an instant messaging store 628,or a social networking site 630. The software applications 420 may useany of these types of systems or the like for enabling data utilization,as described herein. A server 620 may provide the software applications420 to clients. As one example, the server 620 may be a web serverproviding the software applications 420 over the web. The server 620 mayprovide the software applications 420 over the web to clients through anetwork 615. By way of example, the client computing device may beimplemented as the computing device 400 and embodied in a personalcomputer 602 a, a tablet computer 602 b, and/or a mobile computingdevice (e.g., a smart phone) 602 c. Any of these embodiments of theclient device may obtain content from the store 616.

The description and illustration of one or more embodiments provided inthis application are intended to provide a complete thorough andcomplete disclosure the full scope of the subject matter to thoseskilled in the art and not intended to limit or restrict the scope ofthe invention as claimed in any way. The embodiments, examples, anddetails provided in this application are considered sufficient to conveypossession and enable those skilled in the art to practice the best modeof claimed invention. Descriptions of structures, resources, operations,and acts considered well-known to those skilled in the art may be briefor omitted to avoid obscuring lesser known or unique aspects of thesubject matter of this application. The claimed invention should not beconstrued as being limited to any embodiment, example, or detailprovided in this application unless expressly stated herein. Regardlessof whether shown or described collectively or separately, the variousfeatures (both structural and methodological) are intended to beselectively included or omitted to produce an embodiment with aparticular set of features. Further, any or all of the functions andacts shown or described may be performed in any order or concurrently.Having been provided with the description and illustration of thepresent application, one skilled in the art may envision variations,modifications, and alternate embodiments falling within the spirit ofthe broader aspects of the general inventive concept embodied in thisapplication that do not depart from the broader scope of the claimedinvention.

What is claimed is:
 1. A method allowing computer-generated speech to berendered with a multi-voice font having different speakercharacteristics and/or prosody than source voice fonts used to generatethe multi-voice font, the method comprising the acts of: loading thesource voice fonts; assigning weights to characteristics of each sourcevoice font relating speaker characteristics or prosody; obtaining textto be rendered as the computer-generated speech; predictingcharacteristic values for the text using each source voice font; andmerging the predicted characteristic values with the correspondingweights to produce interpolated characteristic values ready to be usedfor rendering the text as computer-generated speech.
 2. The method ofclaim 1 wherein the act of merging the predicted characteristic valueswith the corresponding weights further comprises the acts of:multiplying the predicted characteristic values by the weight for thecharacteristic given to the source voice font used to predict thepredicted characteristic values; and summing the weighted characteristicvalues to produce the interpolated characteristic values.
 3. The methodof claim 1 wherein the act of assigning weights to characteristics ofeach source voice font further comprises the acts of: assigning aduration weight to each source voice font; assigning a f0 weight to eachsource voice font; and assigning a spectrum weight to each source voicefont.
 4. The method of claim 3 wherein the act of predictingcharacteristic values for the text using each source voice font furthercomprises the act of predicting duration values, voiced/unvoicedprobability values, f0 values, and spectral trajectory values for thetext using each source voice font.
 5. The method of claim 3 wherein theact of obtaining text to be rendered as the computer-generated speechfurther comprises the act of parsing the text into a sequence ofphonemes dividable into frames.
 6. The method of claim 5 wherein the actof predicting characteristic values for the text using each source voicefont further comprises the acts of: predicting a duration value and avoiced/unvoiced probability value for each phoneme using each sourcevoice font; and predicting a f0 value and a spectral trajectory valuefor each frame of each phoneme using each source voice font.
 7. Themethod of claim 5 wherein the act of merging the predictedcharacteristic values with the corresponding weights further comprisesthe acts of: interpolating a duration value for each phoneme using theduration weights; interpolating an voiced/unvoiced probability value foreach phoneme using the spectrum weights; making a voiced/unvoiceddecision for each phoneme based on the voiced/unvoiced probability valuefor that phoneme; interpolating a f0 value for each phoneme using the f0weight; normalizing the f0 values; and interpolating a spectraltrajectory value for each phoneme using the spectrum weights.
 8. Themethod of claim 7 wherein: the act of interpolating a duration value foreach phoneme using the duration weights further comprises the acts of:multiplying the predicted duration values predicted using each sourcevoice font by the corresponding duration weight to produce weightedduration values; and summing the weighted duration values from eachsource voice font for each phoneme to produce an interpolated durationvalue for that phoneme; the act of interpolating a voiced/unvoicedprobability value for each phoneme using the spectrum weights furthercomprises the acts of: multiplying the predicted voiced/unvoicedprobability values predicted using each source voice font by thecorresponding spectrum weight to produce weighted voiced/unvoicedprobability values; and summing the weighted voiced/unvoiced probabilityvalues from each source voice font for each phoneme to produce aninterpolated voiced/unvoiced probability value for that phoneme; the actof interpolating a f0 value for each phoneme using the f0 weightsfurther comprises the acts of: multiplying the predicted f0 valuespredicted using each source voice font by the corresponding f0 weight toproduce weighted f0 values; and summing the corresponding weighted f0values from each source voice font for each frame to produce aninterpolated f0 value for that frame; and the act of interpolatingspectral trajectory value for each phoneme using the spectrum weightsfurther comprises the acts of: multiplying the predicted spectraltrajectory values predicted using each source voice font by thecorresponding spectrum weight to produce weighted spectral trajectoryvalues; and summing the corresponding weighted spectral trajectoryvalues from each source voice font for each frame to produce aninterpolated spectral trajectory value for that frame.
 9. The method ofclaim 1 further comprising the act of synthesizing the text ascomputer-generated speech using the interpolated characteristic values.10. The method of claim 1 further comprising the act of storing amulti-voice font definition specifying the source voice fonts used togenerate the multi-voice font and linking each source voice fonts withthe characteristic weights assigned to that source voice font.
 11. Asystem for generating a multi-voice font from a plurality of sourcevoice fonts, the system comprising: a phoneme sequencer for parsinginput text into a sequence of phonemes; a predictor operable to predictvalues of voice font characteristics for the phonemes usingcharacteristic models for a plurality of source voice fonts; a weightselector operable to assign a duration weight, a f0 weight, and aspectrum weight to each source voice font, the duration weight, the f0weight, and the spectrum weight determining the relative contribution ofthe voice font characteristics predicted for the corresponding sourcevoice font to the multi-voice font; and an interpolator operable tomerge the predicted voice font characteristics with the weights toproduce the multi-voice font having voice font characteristics derivedfrom the source voice fonts.
 12. The system of claim 11 wherein thepredictor further comprises: a duration predictor operable to predictduration values for each phoneme using a duration prediction modelprovided by each source voice font; a f0 predictor operable to predictf0 values for each phoneme using a f0 prediction model provided by eachsource voice font; a spectral trajectory predictor operable to predictspectral trajectory values for each phoneme using a spectrum predictionmodel provided by each source voice font; and a voiced/unvoicedprobability predictor operable to predict voiced/unvoiced probabilityvalues for each phoneme using a voiced/unvoiced decision model providedby each source voice font.
 13. The system of claim 11 wherein theinterpolator further comprises: a duration interpolator operable tomerge the predicted duration values for each phoneme with the durationweight for the predicting source voice font to produce an interpolatedduration value for each phoneme; a f0 interpolator operable to merge thepredicted f0 values for each phoneme with the f0 weight for thepredicting source voice font to produce an interpolated f0 value foreach frame of the phoneme; a spectral trajectory interpolator operableto merge the predicted spectral trajectory values for each phoneme withthe spectrum weight for the predicting source voice font to produce aninterpolated spectrum trajectory value for each frame of the phoneme;and a voiced/unvoiced decision interpolator operable to merge thepredicted voiced/unvoiced probability values for each phoneme with thespectrum weight for the predicting source voice font to produce aninterpolated voiced/unvoiced probability value for each phoneme andcompare the interpolated voiced/unvoiced probability value to athreshold to determine an interpolated voiced/unvoiced decision valuefor each phoneme.
 14. The system of claim 11 further comprising anormalizer operable merge the spectrum weight with an estimated f0 upperlimit and an estimated f0 lower limit for the predicted f0 range foreach frame of each phoneme to produce a first f0 limit pair, merge thef0 weight with an estimated f0 upper limit and an estimated f0 lowerlimit for the interpolated f0 range for each frame of each phoneme toproduce a second f0 limit pair, and normalize the interpolated f0 valuesusing the first f0 limit pair and the second f0 limit pair.
 15. Acomputer readable medium containing computer executable instructionswhich, when executed by a computer, perform a method of generating amulti-voice font for rendering text as computer-generated speech with,the method comprising the acts of: obtaining the text to be rendered asthe computer-generated speech; loading the source voice fonts;predicting duration values, voiced/unvoiced probability values, f0values, and spectral trajectory values for the text using each sourcevoice font; assigning a duration weight, a f0 weight, a spectrum weightto each source voice font; merging the duration values predicted witheach source voice font with the duration weight to produce interpolatedduration values, the duration weight for each source voice fontrepresenting the percentage that the source voice font contributes tothe interpolated duration values; merging the f0 values predicted witheach source voice font with the f0 weight given to that source voicefont to produce interpolated f0 values, the f0 weight for each sourcevoice font representing the percentage that the source voice fontcontributes to the interpolated f0 values; and merging thevoiced/unvoiced decision values and the spectral trajectory valuespredicted with each source voice font with the spectrum weight given tothat source voice font to produce interpolated voiced/unvoicedprobability values and interpolated spectral trajectory values, thespectrum weight for each source voice font representing the percentagethat the source voice font contributes to the interpolatedvoiced/unvoiced probability values and interpolated spectral trajectoryvalues.
 16. The computer readable medium of claim 15 wherein the methodperformed by the computer executable instructions further comprises theact of using the interpolated duration values, the interpolatedvoiced/unvoiced decision values, the interpolated f0 values, and theinterpolated spectral trajectory values to render the text ascomputer-generated speech.
 17. The computer readable medium of claim 15wherein the method performed by the computer executable instructionsfurther comprises the act of parsing the text into a sequence ofphonemes with each phoneme dividable into frames and: the act of mergingthe duration values predicted with each source voice font with theduration weight given to that source voice font to produce interpolatedduration values further comprises the act of, for each phoneme, summingthe products of the duration value predicted by each source voice fontand the corresponding duration weight; the act of merging the f0 valuespredicted with each source voice font with the f0 weight given to thatsource voice font to produce interpolated f0 values further comprisesthe acts of: for each frame, summing the products of the f0 valuespredicted by each source voice font and the corresponding f0 weight; andnormalizing the interpolated f0 values; and the act of merging thevoiced/unvoiced probability values and the spectral trajectory valuespredicted with each source voice font with the spectrum weight given tothat source voice font to produce interpolated voiced/unvoicedprobability values and interpolated spectral trajectory values furthercomprises the acts of: for each phoneme, summing the products of thevoiced/unvoiced probability values predicted by each source voice fontand the corresponding spectrum weight; determining whether each phonemeis voiced using the interpolated voiced/unvoiced probability value foreach phoneme; and for each frame, summing the products of the spectraltrajectory values predicted by each source voice font and thecorresponding spectrum weight.
 18. The method of claim 1 wherein the actof assigning weights to characteristics of each source voice fontfurther comprises the act of proportioning the weights for eachcharacteristic such that the sum of the weights for each characteristicis substantially equal to one.
 19. The computer readable medium of claim15 wherein the method performed by the computer executable instructionsfurther comprises the act of synthesizing the text as computer-generatedspeech using the interpolated characteristic values.
 20. The computerreadable medium of claim 15 wherein the method performed by the computerexecutable instructions further comprises the act of storing amulti-voice font definition specifying the source voice fonts used togenerate the multi-voice font and linking each source voice fonts withthe characteristic weights assigned to that source voice font.